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Measuring the Effects of Problem-Based Learning on the Development of 
Postformal Thinking Skills and Engagement of First-Year Learning 
Community Students 


Abstract 

This article presents an instructional model for teaching a gateway history course that affects students by: l) 
improving their ability to think at higher levels; 2) increasing engagement; and, 3) enhancing their perception 
of the relevancy of the course material in comparison to traditional lecture and discussion. The model includes 
problem-based learning, a metacognitive reflection inducing approach to discussion, and integration into a 
learning community (PBL LC). The researchers conducted an evaluation of the outcomes of PBL LC and 
compared them to the outcomes of the same course taught by model methods without the learning 
community (PBL History) and by traditional lecture and discussion (TLD). We used a neo-Piagetian 
framework for developing the metacognitive reflection approach that also identified our target for higher level 
thinking and adult complex problem solving ability, i.e. postformal thinking dynamics. Change in cognitive 
ability, engagement and perception of relevancy of the content were measured using both quantitative and 
qualitative techniques. The results indicated that students taught with PBL LC had the highest levels of 
postformal thinking change, engagement, and perception of content relevancy, with both PBL LC and PBL 
resulting in higher levels than TLD. 
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Introduction 

Americans have set goals at both national and state levels to make America 
the country with the largest percentage, 60%, of its population having post¬ 
secondary degrees. America is a long way from achieving that goal with 38% of 
Americans having achieved that status by the end of 2008. During that same year, 
fewer than 41% of 18-24 year old Americans were enrolled in 2- or 4-year degree 
programs. The numbers, which varied dramatically by gender and ethnicity, 
resulted from both low matriculation and low year-to-year retention of students in 
school, particularly for minorities. Lack of educational attainment has the 
potential to jeopardize the future prosperity of Americans (Liu, 2011). Increased 
engagement and the perception of content relevance are important for academic 
success, including year-to-year retention, particularly for students who differ from 
the majority group (Kahu, 2013). 

The need to improve cognitive skills has also been identified. The 
Association for American Colleges and Universities (AAC&U) published College 
Learning for the New Global Century (2007) to identify essential learning 
outcomes and guiding principles for a twenty-first-century college education. The 
report called for curricular and pedagogical structures that equip students with the 
cognitive skills necessary to understand and respond to the complex challenges of 
modern life. These skills include inquiry and analysis, critical and creative 
thinking, teamwork and problem solving “practiced extensively, across the 
curriculum, in the context of progressively more challenging problems, projects, 
and standards for performance” (p. 3). 

Our teaching and research have been directed towards addressing these 
issues. We have designed and tested a problem-based instructional model to 
determine the extent to which it improves cognitive skills, engagement, and the 
perception of content relevancy, particularly for first-year students in gateway or 
survey level courses. 


Literature Review 


Postformal Thinking 

Sinnott’s (1998) work provided our framework for measuring the effects of 
curricular variations on thinking skills. We believe this is a useful frame because 
it provides the opportunity to measure thinking levels within both Piagetian and 
neo-Piagetian categories. However, not much research based on principles of 
cognitive psychology has been conducted to explore the effects of using problem 
based learning in college classrooms, and more specifically, within learning 
communities. 
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Sinnott (1998) asserted that the university should meet the demands of a 
changing world by approaching the task as one of enhancing postformal thinking 
or adult reasoning skills in its students. Sinnott’s argument is based, in part, on the 
assumption that most college students lack postformal thinking skills and rely on 
inadequate thinking systems to address complex problems and issues. Other 
theorists present similar arguments (Demetriou, Spanoudis, & Mouyi, 2011; 
Kramer, 1983; Kramer, Kahlbaugh, & Goldston, 1992; Perry, 1970, 1981, 1999). 

Our problem-based instructional model is designed to facilitate the 
following potential theoretical outcomes related to postformal thinking and adult 
reasoning: 

(1) recognition and utilization of multiple cognitive operations (intuitive, 
formal, relativistic, dialectical, and domain specific processes, i.e. historical 
thinking), and that the choice of logical systems of thinking is subjective and 
based on the type of problem encountered; (2) recognition that the social 
context of learning and problem-solving will often lead to group selection of 
the cognitive operations applied to a specific problem, which often leads to a 
collective cognition which, in itself, is a form of social learning or cognitive 
scaffolding (Sinnott, 1998); (3) application of knowledge and skills to 
construct deeper and expanded insights of course content; and (4) 
recognition and utilization of epistemic/metacognitive reflective skills. 
(Wynn, 2010, p. 10) 

Engagement 

Pascarella and Terenzini (2005) observed that the importance of student 
engagement in encouraging intellectual and academic success has been known for 
a long time. The construct has been defined and operationalized in a number of 
ways, including views that emphasize behavioral, cognitive and emotional aspects 
of student performance (Kahu, 2013). Kahu observed that students often 
experience something akin to “culture shock” when they first start college. This 
can happen more often with students who are different from the majority group 
and who arrive at college, as she describes it, “not having the necessary social, 
cultural, and academic capital to fit into the university culture” (p. 763). She 
speculates that this perspective can provide explanations for why students are 
engaged or alienated, and posits that educational institutions should take a holistic 
perspective on engagement by developing processes for “engaging students” with 
the outcome being “engaged students” (p. 764). 

Student engagement has been measured at both the institutional and 
classroom levels. For instance, the National Survey of Student Engagement 
(NSSE) is a widely used and well-known measure at the institutional level. 
Student engagement in the classroom has been shown to be related to student 
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retention from year to year, persistence to graduation, and success, as measured 
by increased GPA (Handelsman, Briggs, Sullivan & Towler, 2005; Mosholder & 
Tolman, 2012; Mosholder, in press). Similarly, problem-based learning (PBL) has 
been shown to promote cognitive engagement (Rotgans & Schmidt, 2011). 

Instructional Interventions 

Intervention Timing 

The timing of instructional interventions is an important consideration. The 
first year of college is a critical curricular focal point in guiding students toward 
developing critical thinking skills and encouraging them to stay in school. 
Moreover, potential yet critical gains in learning and cognitive development are 
associated with the transitional period of late-adolescence (Baxter Magolda, 2009; 
Parks, 2000; Pascarella, 2005; Reason, Terenzini, & Domingo, 2006; Tanner, 
Arnett, & Leis, 2008; Wynn, 2010). 

Late adolescents often utilize a dual process model of cognition when 
problem-solving (Keating, 2004; Witteman, van den Bercken, Claes, & Godoy, 
2009). Intuitive thinking is the first mode, governed by an “if it feels right, it’s 
right” approach that operates in a more automatic, holistic manner. Rational or 
formal thinking, the second mode, is characterized as slower, more deliberative 
and rule-governed operations (Epstein, Pacini, Denes-Raj, & Heier, 1996; Evans, 
2008; Witteman et. al, 2009). Even if late adolescents pull away from intuitive 
thinking to apply more ordered formal thinking in their attempt to solve a 
complex problem or issue, they often do so through a closed systems approach 
(Sinnott, 1998). A closed systems problem-solver will generally apply a practiced 
systematic/formal problem-solving framework based on previous experience with 
similar problems. 

This closed systems framework typically involves a problem-solving 
dynamic based on a limited number of variables, with other important aspects of 
the problem often judged as irrelevant to the solution. Formal thinkers often 
expect to produce a single right answer that will apply to all similar circumstances 
(Wu & Chiou, 2008). Late adolescents must come to recognize the inadequacy of 
a formal/closed systems approach to problem solving to prompt a search for more 
adequate systems to solve more complex problems (Sinnott, 1998). 

For the neo-Piagetians the ideal transitional dynamic of late adolescent 
cognition involves moving from a formal/closed systems problem-solving 
approach towards a relativistic thinking system, the first level of postformal 
thinking (Chiou, 2008; Kahlbaugh & Kramer, 1995; Kramer et al., 1992). 
Relativistic thinkers recognize that when a person’s perspective or context 
changes, her/his perspective on what is true also changes (Chiou, 2008). 
Kahlbaugh and Kramer (1995) link the transition to relativistic thinking to the 
multiple challenges and complexities faced by late-adolescents and young adults. 
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Rather than seeing problems and issues through the lens of fixed truths—good 
versus bad, for example—relativistic thinkers recognize that context and 
contradictions are key to understanding the complexities of a problem and to 
developing potential resolution alternatives. Within this frame, relativistic 
thinkers may come to recognize that for some problems and issues no resolution 
may be possible (Chiou, 2008; Sinnott, 1998). 

The most advanced cognitive processes associated with adult problem 
solving recognized by the neo-Piagetians occur within dialectical thinking, the 
second and final stage of postformal thinking (Basseches, 1984, 1989; Sinnott 
1998). Kramer et al. (1992) describe dialectical thinking as the integration of 
relativistic thinking with the recognition that both sides of contradictions within a 
problem or issue are interrelated and connected, and are critical in the 
development of resolution alternatives. Inconsistencies and contradictions within 
problems and issues become catalysts in the application of multiple cognitive 
systems as dialectical thinkers seek resolutions that lead to higher levels of 
understanding and cognition (Ho, 2000). Dialectical thinkers also recognize that 
any resolution or stability that may result from dialectical problem-solving 
operations will be perpetually challenged by new challenges, changes, and a 
potential tension to resolution to tension dynamic (Blouin & McKelvie, 2012). 

Learning Communities 

One instructional intervention that has the ability to address these challenges 
is learning communities, programs explicitly designed to provide opportunities for 
students to practice integrative and interdisciplinary learning, fostering the 
development of the types of advanced cognitive skills called for in the AAC&U 
Report. The National Resource Center for Learning Communities (Washington 
Center, n.d.a) describes learning communities as classes that are linked or 
clustered, often around an interdisciplinary theme, and that enroll a common 
cohort of students, with the goals of increasing student involvement and 
motivation and enhancing student intellectual development. Participation in a 
learning community has been positively related to the development of higher 
order thinking and problem-solving skills (Rocconi, 2011; Pike, 1999; Zhao & 
Kuh, 2004). 

The general structure of a learning community offers an ideal environment 
to nurture and support first-year college students toward the type of meaningful 
learning and cognitive advancement described above. The National Resource 
Center for Learning Communities (Washington Center, n.d.b) advocates learning 
environments within a learning community context that foster “the habits of mind 
and skills to tackle complex real-world issues” (para. 1). Thus, there is great 
potential for epistemological alignment within first-year learning communities if 
courses are constructed around two primary bases: the cognitive needs and nature 
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of the late adolescent, and instructional methods that promote deep meaning and 
cognitive growth. We believe problem-based learning activities with a 
metacognitive reflection process will provide scaffolding for first-year students to 
develop these advanced cognitive skills, specifically, postformal operations. 

Problem-Based Learning 

Originally developed for medical education, problem-based learning (PBL) 
is a flexible instructional strategy in which students are guided to take part in the 
cognitive processes of advanced problem solving (Lenkauskaite & Mazeikiene, 
2012). It has subsequently been used in a variety of educational settings, from 
middle and secondary education to higher education (Hmelo-Silver, 2004). 
Pascarella (2005) asserts that problem-based instructional approaches help 
facilitate cognitive growth during the late adolescent transitional period by 
prompting student learning in a constructivist and socially interactive context. 

Hung (2013) defines PBL as “an instructional method aimed at preparing 
students for real-world settings” (p. 31). “By requiring students to solve problems 
as the main format of instruction, PBL enhances students’ learning outcomes by 
promoting their abilities and skills in applying knowledge, solving problems, 
practicing higher order thinking, and self-directing and reflecting on their own 
learning...” (p. 31). 

The question of whether or not PBL is more effective than traditional 
methods has been addressed through several meta-analyses (Albanese & Mitchell, 
1993; Strobel & van Barneveld, 2009; Vernon & Blake, 1993; Walker & Leary, 
2009), with some studies indicating little or no beneficial effect on student 
learning of content/concepts. However, the analysis of later studies involving 
multiple disciplines outside medical education (Gijbels, Dochy, Van den Bossche, 
& Segers, 2005) has shown that PBL students did as well as or better than their 
lecture-based counterparts (Walker & Leary, 2009). Strobel and van Barneveld 
(2009) note that results indicated that “PBL is significantly more effective than 
traditional instruction to train competent and skilled practitioners and to promote 
long-term retention of knowledge and skills acquired during the learning 
experience” (p. 55). Several researchers have argued that PBL facilitates students’ 
content knowledge retention and enhances their ability to successfully apply 
problem-solving strategies in new and complex situations (Blumberg, 2000; 
Cognition and Technology Group at Vanderbilt [CTVG], 1997; Maxwell, 
Bellissimo, & Mergendoller, 2001; Mergendoller, Maxwell, & Bellissimo, 2006). 

Facilitating Cognitive Growth through PBL 

Wynn (2010, in press) has developed a metacognitive reflection process 
within our PBL model that is designed to promote cognitive growth and advanced 
problem-solving skills through instructor and peer modeling. When used within a 


5 



Learning Communities Research and Practice, Vol. 2 [2014], Iss. 2, Art. 4 


collaborative learning environment, this modeling and cognitive scaffolding 
prompts more capable peers to accomplish a tutorial role similar to that of the 
instructor. Vygotsky’s (1978) social learning theory describes this relationship 
between instructors, tutors and learners in the context of problem-solving through 
the developmental concept of the zone of proximal development (ZPD). Vygotsky 
defines ZPD as “the distance between the actual developmental level as 
determined by independent problem solving and the level of potential 
development as determined through problem solving under adult guidance or in 
collaboration with more capable peers” (p. 86). 

Problem-based learning with this metacognitive reflection process built in 
provides multi-systematic scaffolding, a combined epistemic and metacognitive 
process: the instructor prompts students to reflect on the dynamics of the 
reasoning skills (e.g., intuitive, formal, relativistic, and dialectical) that they 
practice under his or her guidance during problem-solving, and subsequently to 
judge which systems were more useful or successful and why (Hmelo-Silver, 
2004; Vukman, 2005; Wynn, 2010). Over time, this helps students develop a 
cognitive self-awareness that is valuable in the development of their postformal 
thinking/problem-solving skills as well as their ability to monitor and direct the 
processes of problem-solving in general (Hacker, 1998; Hmelo-Silver, 2004; 
Wynn, 2010). 

Without opportunities for cognitive guidance and reflection in a PBL 
context, when closed systems problem-solvers in the classroom are confronted 
with a complex problem/issue, they may seek to maintain their cognitive 
equilibrium by sealing themselves off from different viewpoints or by rejecting 
formal thinking and utilizing less logical forms of thought like intuitive thinking. 
Thus, a PBL instructional model that explicitly prompts metacognitive reflection 
allows instructors to guide students’ transition from formal/absolutist thinking 
toward the practice and development of postformal thinking by exposing them to 
the diverse perspectives, multiple truths, and contradictions inherent in complex 
problems and issues, and by providing them opportunities for careful, critical 
reflections as part of a modeling process (Basseches, 2005). 

The effect of PBL and metacognitive reflection on the development of these 
cognitive skills among first-year college students and college students in general 
has not been studied. Therefore, based on the above literature review and our own 
observations of PBL and first-year learning community learning environments, 
our research tested the following hypotheses. 

1. There will be no significant difference in cognitive growth, as measured 
via pretest-posttest changes in postformal thinking skills, between 
students in a PBL American history-based first-year learning community 
and a PBL regular (non-learning community) section of the same course. 
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2. There will be no significant difference in cognitive growth, as measured 
via pretest-posttest changes in postformal thinking skills, between 
students in a PBL regular section of American history and the same 
course taught through primarily lecture/discussion. 

3. There will be no significant difference in self-reported level of course 
engagement, as measured via an end of course questionnaire, between 
students in a PBL American history-based first-year learning community 
and a PBL regular (non-learning community) section of the same course. 

4. There will be no significant difference in self-reported level of course 
engagement, as measured via an end of course questionnaire, between 
students in a PBL regular section of American history and the same 
course taught through primarily lecture/discussion. 

5. There will be no significant difference in self-reported level of course 
content relevance, as measured via an end of course questionnaire, 
between students in a PBL American history-based first-year learning 
community and a PBL regular (non-learning community) section of the 
same course. 

6. There will be no significant difference in self-reported level of course 
content relevance, as measured via an end of course questionnaire, 
between students in a PBL regular section of American history and the 
same course taught through primarily lecture/discussion. 

Method 

We evaluated our PBL instructional model in a gateway history course, 
America since 1890, combined with a first-year learning community (LC) under 
the theme, “Stepping into America’s Past: What Would You Do?” (PBL LC). 
Two of these LCs, with 25 students in each, were included in our study, with one 
of the researchers teaching both LC history sections and one of the other 
researchers teaching the linked first-year seminar sections. We included two other 
curricular variables in our study: PBL instruction within a non-LC section (PBL 
History) that enrolled 40 students, and traditional lecture and discussion (TLD) 
within three non-LC sections that enrolled 50 students per section. One of the 
researchers developed and implemented six problem/issue-based activities in each 
of the three PBL sections of American history. Appendix A includes the course 
topical/unit outline with a list of the PBL activities. Each PBL activity took 
between one and three 75-minutes class periods to complete. 
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In addition to the PBL activities, the researcher teaching the three PBL 
history courses used lecture, discussion, and guided questions (Riseman & 
Wineberg, 2008), with a focus on establishing historical context for the PBL 
activities. Therefore, the full instructional dynamics of the course may be 
described as hybridized (Allen, Donham, & Bernhardt, 2011) or “problem- 
assisted learning” rather than being exclusively or purely PBL (Murray & 
Summerlee, 2007; Chapman, Keller, & Fournier, 2002). A broad 
progressive/conservative dialectical framework was also used throughout the 
course to examine/analyze American history during the period studied (1890 to 
the present). 

The learning objectives of the first-year seminar linked to the two sections 
of the PBL American History courses included strategies for academic success, 
life and motivational skills, and the foundations for global learning. The 
researcher who taught this course also created lesson plans to focus skill 
development on success in the American history curriculum. In addition, students 
in the first-year seminar were provided opportunities to work on PBL-related 
assignments and to debrief and reflect upon the thinking systems they practiced in 
the PBL activities. 

The instructor for the third curricular variable (TLD) taught the three other 
sections of the same American history course. This instructor primarily used the 
traditional lecture method. In addition, he conducted four in-class exercises using 
the Taking Sides series, for which students read scholarly essays that presented 
alternative yes or no viewpoints as well as supporting arguments about historical 
events. The instructor then asked the students to write an essay outlining and 
supporting their opinion on one of these issues. 

Participants were included in the study based on their enrollment in the 
courses outlined above and their consent. The number of participants was as 
follows: PBL LC = 40, PBL History = 31, TLD = 35. 

Details of Our PBL Model 

Dolmans and Gijbels (2013) note a lack of explicit descriptions of PBL 
instructional models in published studies. To help address that void, we note that 
three cognitive-based instructional principles (Driscoll, 1994) guided the 
development of the PBL instructional model evaluated in this study: 

1. The learning environment should be active and discovery-oriented. 

2. Peer learning and social negotiation should be prominent to encourage 
cognitive development. 

3. Problem solving and Socratic dialogue and similar instructional 
strategies should be used to promote cognitive development. (Downing, 
Kwong, Chan, Lam, & Downing, 2009, p. 619) 
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The first two principles are operationalized in Step 1 of the instructional 
model as described below. The third principle is operationalized in Steps 2 and 3, 
with Step 2 including a decision-based/argumentation structure (Jonassen, 2012). 

The PBL procedures are generally based on Edens’ (2000) three-phase PBL 
model and were adapted from Wynn’s (in press) PBL instructional model. 

Step 1 - Introduction of the Problem: The primary focus in Step 1 was 
to pique student interest (create a need to know more), establish 
“stakeholdership,” and explicitly portray the problem/issue as 
multidimensional. Problems/issues were introduced through story¬ 
telling, video clips, data presentations, and readings. Students were 
guided to recognize that the problem/issue has more than one “right” 
answer and opposing positions. For example, the introduction of the 
first PBL activity in the PBL sections of the American history course 
in the current study involved sharing a story (with illustrations) of a 
U.S. merchant ship encountering an uncharted Pacific island that could 
be very useful as a U.S. colony. The class then identified how different 
groups—i.e., U.S. expansionists, U.S. anti-expansionists, the natives 
on the island, other colonial powers—would see this encounter. 
Students were then able to identify the issue/problem to be resolved 
as: Should the U.S. annex the island as a colony? 

Step 2 - Initiation of PBL Events-Argumentation and Student Inquiry: 

Step two included a decision-based/argumentation structure (Jonassen, 

2012) in which students generated arguments and worked to recognize 
conflicts and contradictions among competing positions. This was 
done primarily through historical simulations and current issue 
presentations at the end of the course. For example, the first PBL 
activity included a U.S. Senate Sub-Committee Hearing on U.S. 
Expansion. The class was divided into Expansionists, Anti- 
Expansionists, and Senators in order to outline a rationale and gain 
support for their assigned position on the problem/issue. Senators 
ultimately voted on the status of the newly discovered Pacific island 
after hearing both arguments (to take or not take the island). At the 
conclusion of the simulation, students identified both what they had 
learned about the problem/issue and the inherent contradictory or 
opposing positions and what additional information they needed in 
order to develop potential alternatives/solutions to the issue of U.S. 
expansion. 
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Step 3 - Problem Solution: Students generated solutions/decisions, 
examined their “fit,” and then proposed the most appropriate one and 
evaluated its historical or potential consequences. For example, after 
the vote in Senate subcommittee hearing, students used the 
contradictions identified between the opposing positions and their 
understanding of the context of the issue applied to the actual decision 
by the U.S. to annex the Philippines after the Spanish-American War. 
Students then constructed a solution alternative, deciding that the 
Philippines should be offered territorial rather than colonial status in 
1899, with a timeline for independence established by U.S. and 
Filipino representatives. This solution was compared to the actual 
outcome of U.S. annexation of the Philippines in the short and long 
term. A concluding opinion essay was assigned at the end of the 
activity, followed by a debriefing session that included a review of the 
content, concepts, and skills encountered and used during the activity. 

Step 3 ended with a metacognitive reflection questionnaire (Appendix B) 
that provided a guided reflection on the successes and failures of each of the 
thinking strategies utilized by students (Wynn, 2010; Wynn, in press). For 
example, as students reflected on their thinking during Step 2 of the PBL activity, 
many recognized that their efforts to persuade the Senators to vote for their 
assigned positions (Expansionists or Anti-Expansionists) had prompted 
intuitive/emotional thinking along with logical analysis, and some recognized that 
intuitive/emotional thinking made it harder for them to consider the validity of the 
opposing position. Several Senators in the activity recognized they were utilizing 
more relativistic thinking as they considered the complexities of the issue, 
identifying or empathizing with both sides and multiple perspectives as they 
constructed a rationale for their vote during Step 2. Students also noted they had 
been prompted to practice relativistic and even dialectical thinking during Step 3 
of the activity, in that they utilized contradictions they identified as inherent in the 
issue—as well as the multiple perspectives involved—in the process of 
constructing and selecting a solution alternative. 

Measures 

We used the Postformal Thought (PFT) Questionnaire (Sinnott & Johnson, 
1997) to measure participants’ level of postformal thought. We administered the 
PFT on both the first and the last day of classes. The questionnaire included 10 
statements representing different operations of postformal thinking. Participants 
responded to each statement by indicating the extent to which it characterized 
their own thinking (7=very true to l=not true). 


http://washingtoncenter.evergreen.edu/lcipjournal/vol2/iss2/4 


10 



Wynn et al.: Measuring the Effects of Problem-Based Learning 


Cartwright, Galupo, Tyree, and Jennings (2009) tested the reliability and 
construct validity of the PFT and found it to be a moderately reliable (.63) and 
valid measure of postformal thought. For this study, scores were summed for the 
10 items for each participant. Potential scores range from 10, indicating low levels 
of postformal thought, to 70, indicating high levels of postformal thought 
(Cartwright et al., 2009). The PFT is included as Appendix C. 

An end of study questionnaire (ESQ) was administered to participants after 
all other data were collected. The ESQ included five questions, two of which 
called for a Likert rating on their level of engagement in the history course 
(Questions 1 and 2), one that called for a Likert rating on the level of relevance of 
course content and topics (Question 3), and two that prompted participants to 
reflect on the extent to which their experience in the history course expanded their 
ability to think critically (Questions 4 and 5). The ESQ is included as Appendix D. 

Methods of Analysis 

We measured the development of postformal thinking skills, engagement, and 
perception of content relevancy with each of the curricular variables using both 
quantitative and qualitative methods. A one-way ANOVA was used to analyze data 
from the ordinal variables. We used directed content analysis to analyze Questions 
four and five of the ESQ (Hsieh & Shannon, 2005). Researchers use directed 
content analysis when “prior research exists about a phenomenon that...would 
benefit from further description” (p. 1281). This approach has also been referred to 
as deductive category application (Mayring, 2000). Using the literature review as a 
starting point, two of the researchers identified key concepts and variables of four 
categories of problem-solving thinking systems to be used as initial coding 
categories: 1) intuitive, 2) analytic, 3) relativistic, and 4) dialectic. These categories 
proved difficult to use in practice, so they were collapsed into operational definitions 
for two problem-solving systems: closed systems (CS) and postformal operational 
(PF) (Potter & Levine-Donnerstein, 1999). These operational definitions are 
presented in Appendix E. 

Student responses were coded as self-reported PFT experience if the comment 
indicated the utilization of characteristics of relativistic and/or dialectical operations. 
The results were given to the third researcher for statistical analysis. A 75 percent 
agreement was necessary for a comment to be deemed as PF. A dichotomous 
nominal variable was established (one = self-reported PFT-related experience in 
question four or five, zero = no self-reported PFT-related experience in question 
four or five) in order to cross-tabulate results. Questions four and five of the ESQs 
were coded individually by two of the researchers and two student researchers. 
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Results 

Postformal Thinking Skill Development 

The results of the pre- and post-administrations of the PFT are shown in 
Tables 1 and 2. Table 1 shows the mean difference between time 2 and time 1 in 
the same units as the Likert scale of the instrument, i.e., 1 through 7. Table 2 
shows the normalized gain score, which is a measure of potential gain on a scale 
of 0 to 1, with 1 representing all possible gain (Bao, 2006). Hake (1998) defines 
the normalized gain score as “the ratio of the actual average gain (%<post> - 
%<pre>) to the maximum possible average gain (100-%<pre>)” (p. 64). 


Table 1 

One-way ANOVA Data of the Net Postformal Gain Scores of Students in the Three 
Curricular Variables 


Variable 

df 

Mean 

F 

Sig. 

PBL LC 

2 

4.25 

4.23 

.017 


103 




PBL History 


2.71 



TL 


0.29 




Table 2 

One-way ANOVA Data of the Normalized Postformal Gain Scores of Students in the 
Three Curricular Variables 


Variable 

df 

Mean 

F 

Sig. 

PBL LC 

2 

.094 

4.56 

.013 


103 




PBL History 


.060 



TL 


.008 




As can be seen from these data, increases in post-formal thinking ability 
occurred, with the PBL LC group having significantly greater gain than either the 
PBL History or the TLD group. Therefore, we reject our first hypothesis. In 
addition, the PBL History group had a significantly greater increase than the TLD 
group, leading us to reject Hypothesis 2. 

The results of the directed content analysis of questions 4 and 5 of the ESQ 
are presented in Table 3. The pattern observed with the pre- and post¬ 
administration of the PFT is observed with this data as well, with the greatest 
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frequency occurring in the PBL LC group and the smallest frequency occurring in 
the TLD group, which also supports the rejection Hypothesis 1 and Hypothesis 2. 

Table 3 

One-way ANOYA Data of Responses to Questions Four and Five on the End of Study 
Questionnaire 


Variable 

df 

Mean 

F 

Sig. 

PBL LC 

2 

97 

0.95 

13.6 

.000 

PBL History 


0.72 



TL 


0.43 




Note. One = self-reported postformal thinking experience based on comments from ESQ questions 
four or five; Zero = no self-reported postformal thinking experience based on comments from 
ESQ questions four or five. 

There was a notable difference in the responses of students in the PBL LC 
and PBL History sections on their potential use of the thinking skills they 
practiced, with PBL student comments indicating a higher level of postformal 
thinking. Typical examples from the PBL LC group include: 

Student 107: Yes, I have always considered both sides of a situation, 
but never thought to go in depth on why they have these beliefs. I 
think it will be very helpful with political decisions. The last section of 
current issues made us use the skills we learned throughout the 
semester and apply them to current issues. Now when making 
decisions for who [sic] to vote for and their policies, I have a new 
understanding of the approach. 

Student 130: I believe it has expanded my ability to think critically. I 
have always been a problem solver and understood perspectives as 
well as knowing the facts. I now realize that you have to know 
context, contradictions, multiple solutions, and various perspectives in 
order to effectively solve a problem, and this class confirmed my way 
of thinking and helped me further develop it. 

These statements indicate the use of dialectical thinking, the highest level of 
postformal thinking. Students’ comments from both the PBL LC and PBL History 
were very similar. Two examples are included on the next page. 
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Student 159: I think it’s now easier for me to see things from more than one 
perspective. I understand now that it is important to understand both sides of 
a dilemma or problem before making a decision. Understanding different 
points of view will be very important when dealing with other things in life. 

Student 163: Yes, it’s is easier for me to look at more possible 
solutions and other sides of arguments now. I will definitely continue 
to use these skills. I understand how valuable this skill is now by 
seeing how it helps when trying to make decisions. 

The statements made by students in the TLD sections were, by comparison, 
more indicative of relativistic thinking, the first level of postformal thinking, e.g.: 

Student 209: I definitely have a better grasp of history since the Civil 
War and now. The Taking Sides book encourages looking at both 
sides of the issue. 

Student 264: Yes, I think of the last few presidents very differently; 
knowing what they did for the country. I wrote a paper in this class 
and it taught me how to think both sides of a situation. 

Engagement 

Student responses to the question about course engagement (question one) 
from the ESQ are presented in Table 4. The mean responses follow the same 
pattern observed with postformal thinking development, with PBL LC students 
reporting the highest level of engagement, followed by PBL History students. 
Therefore, we rejected Hypothesis 3 and Hypothesis 4. 

Table 4 

One-way ANOYA Data of Responses to Question One on the End of Study Questionnaire 


Variable 

df 

Mean 

F 

sig 

PBL LC 

2 

4.34 

10.1 

.000 


102 




PBL History 


3.97 



TL 


3.41 




Note: Likert scale with 1 indicating not engaged with the course and 5 indicating fully engaged. 


http://washingtoncenter.evergreen.edU/lcipjournal/vol2/iss2/4 



Wynn et al.: Measuring the Effects of Problem-Based Learning 


Content Relevancy 

Question three on the ESQ asked students to rate the relevancy of the 
content of the course. Again, the same pattern emerges, with the PBL LC students 
responding with significantly higher ratings than the PBL History or TLD and 
with TLD receiving the lowest student rating. The differences with this 
measurement are much less dramatic than those observed with the data reported in 
Tables 1 through 4. However, the difference in means was significant, leading us 
to reject Hypothesis 5 and Hypothesis 6. 

Table 5 

One-way ANOYA Data of Responses to Question Three on the End of Study Questionnaire 


Variable 

df 

Mean 

F 

sig 

PBL LC 

2 

4.77 

3.58 

.032 


100 




PBL History 


4.72 



TL 


4.34 




Note. Likert scale with 1 indicating the course content was irrelevant to the student and 
5 indicating that the content was very relevant. 

Discussion 

The data on the effects of our instructional model, both within a learning 
community and in a non-LC setting, are compelling. Our use of the PBL model 
correlated with greater effects on thinking skills and with greater engagement. In 
addition, students in sections in which the model was used found course content 
to be more relevant. We believe these results have broad implications for teaching 
and learning in general in college gateway/survey courses. However, the 
implications for the first-year LC are even more compelling. 

The significantly greater gain between pretest and posttest means on the 
PLT by the LC group is an indication that a first-year learning community is 
indeed an ideal setting to promote more advanced thinking skills when those skills 
are specifically identified and targeted for cognitive scaffolding. 

We believe several factors help explain the results. Lirst, late adolescence is 
a critical/transitional period for cognitive development (Baxter Magolda, 2009; 
Parks, 2000; Pascarella, 2005; Reason, et al., 2006; Tanner, et al., 2008; Wynn, 
2010). Late adolescent first-year college students may be developmentally suited 
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for the explicit modeling and cognitive scaffolding that occurs through our PBL 
model (Wynn, 2010). 

Second, the two instructors worked closely to integrate topics, including 
building flexibility around the use of class time. If the American history course 
instructor was short on time in completing a PBL activity, students were able to 
work on PBL related assignments in their first-year seminar. The thinking systems 
employed during the activities were also a specific focus in their first-year 
seminar, which addressed the themes of academic success, life and motivational 
skills, and the foundations for global learning. In other words, the thinking 
systems students practiced in their American history course were reinforced in 
their first-year seminar. 

Third, the limit on class size (25 students) and sense of community in our 
LCs may help explain the postformal gain results. Facilitating PBL activities and 
engaging students in the related collaborative/social learning dynamics is simply 
easier and more efficient with 25 students than with 40. Both LC cohorts in our 
study took two classes together, which we believe enhanced classmate familiarity 
and collaboration. This familiarity may have enhanced the PBL environment and 
may also have been a factor in the PBL LC groups’ perspective on level of 
engagement compared to the PBL History group. We were also very encouraged 
by the postformal gains made by the PBL History group compared to the TLD 
group, indicating that our PBL model has a significant effect on the development 
of postformal thinking skills in a non-LC learning environment, even with a larger 
number of students. 

Our PBL model’s impact on student reported level of engagement and 
course relevance was also encouraging. In addition to the class size and sense of 
community factors discussed above, we believe the primary difference in level of 
engagement among students in the PBL History and TLD groups may be 
explained by the explicit learner-centered structure of the PBL instructional 
model, in contrast to the more instructor-centered structure in TLD. Based on past 
course evaluations, both American history instructors may be judged as very 
engaging and dynamic lecturers. However, our instructional model placed 
students at the center of six PBL learning experiences, prompting them to work 
together in a social learning dynamic to define the problem/issue as 
multidimensional, gather relevant information, argue multiple perspectives or 
positions, posit and select solution alternatives, consider how change might affect 
the solutions chosen, and then reflect on the thinking systems they practiced 
during activities. The ESQ question defined engagement as “active participation,” 
which aligned more closely with the social learning dynamics experienced by 
PBL students compared to the more teacher directed lecture/discussion method 
and the Taking Sides activities experienced by the TLD students. 
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We believe the higher level of engagement among PBL students in general 
may have impacted their perception of course content relevance. As well, PBL 
students were confronted with current domestic political/social issues during the 
last unit of their course and developed solution alternatives for these issues, which 
prompted them to relate course content utilized in previous activities and topics to 
issues that more directly affect their lives. 

In summary, we believe the first-year learning community offers an 
epistemologically aligned learning environment to promote cognitive growth, 
specifically, postformal thinking through PBL and metacognitive reflection. Our 
PBL model is explicitly designed to take advantage of the critical period of 
cognitive development associated with the late adolescence and the first year of 
college by guiding students to recognize and practice postformal thinking skills, 
skills that are necessary to effectively confront the complex challenges they will 
face as they continue their academic careers and to effectively deal with the 
complex and often contradictory issues they will inevitably face in their lives in 
general. We are hopeful that our pilot study will provide faculty, staff, and 
administrators in first-year and LC programs with a different and useful 
perspective on the impact of PBL and cognitive scaffolding on the development 
of advanced thinking skills among students, a common general education goal of 
most colleges and universities. 

The Coverage Challenge 

One challenge with PBL and many other learner-centered lessons is that 
they typically take more instructional time than covering the same material in a 
lecture. Weimer (2013), in her essays on learner-centered teaching, points out that 
the desire by faculty to cover all of the content in courses “strongly influences, if 
not dictates, most instructional decisions” (p.46). She suggests learner-centered 
objectives. A “less is more” instructional mindset helps facilitate PBL 
instructional planning toward a deep teaching approach through which students 
construct deeper understandings of content and develop applicable domain 
specific and broader cognitive skills (Wynn, 2010; Wynn, in press). For example, 
rather than a strict chronological coverage approach, the focus of PBL activities in 
all three PBL courses in this study was immersion in key turning points in 
American history. Gateway/survey course instructors are often hesitant to break 
away from a strict coverage model; we hope the results of our study will prove 
persuasive. 


Limitations 

There are a number of threats to internal validity. Intact classes were used, 
which may have led to biased or otherwise differential self-selection. Three 
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different instructors participated in the study, possibly leading to the occurrence 
of uncontrolled extraneous events. Further, our measures rely on students’ self- 
reports, which may have resulted in answering in a perceived socially acceptable 
way. Since the study was conducted in history sections, the work may not be 
generalizable to other subjects. Research participation was low in the three TLD 
sections, with only 35 of 150 students completing useable questionnaires. 
Relatedly, the number of participants in the study in general was too small to 
show significance using multivariate analysis. 

Implications for Practice and Future Research 

Our instructional model was constructed using learning theory and has 
proven, within the limitations of our study, to have greater effects on thinking 
skills, engagement, and content relevance when compared to traditional 
instruction. In order to develop postformal problem-solving skills, students must 
be able to recognize inherent thinking systems. Taking time to implement the 
metacognitive reflection component (questionnaire) of the PBL instructional 
method is critical in guiding students to recognize and reflect upon the extent to 
which they practice multiple thinking systems during PBL activities. We realize 
that instructors might be hesitant to take on the task of guiding the metacognitive 
reflection process. It takes instructional time and some working knowledge of 
thinking systems. The metacogntive reflection questionnaire provided in 
Appendix B is offered as a way to introduce the thinking systems involved in 
problem-solving/decision-making and to facilitate reflective discussions. 
Instructors do not have to be experts to guide the process. As stated above, the 
first-year seminar is an ideal setting to reinforce this reflective process. 

Our future research plans include a second iteration of the current study 
with a larger number of participants in the control group sections of the American 
history course, as well as incentives for completion of all questionnaires. The 
second study will include the forced-choice version of the Social Paradigm Belief 
Inventory (Kramer, et al., 1992), along with the PFT questionnaire, to expand the 
measurement of participants’ level of postformal thought. We also plan to extend 
the application of our mixed method direct content analysis to video recordings of 
all three phases of PBL activities to identify student comments that indicate 
postformal operational characteristics. In addition, we plan to share our results 
with colleagues from other fields who teach survey courses or courses with high 
withdrawal and failure rates with the hope of convincing them to modify their 
approaches in ways that are more student-centered both in terms of methods and 
content. 
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Appendix A 

HIST 2112 PBL LC and PBL History Instructional Units 

Unit 1 - The U.S. as an Empire: Global Power Structure (1890-1905) 

PBL Activity - The Question of U.S. Expansion: Expansionists versus 
Anti-Expansionists 

Unit 2 - Social and Political Dynamics in the Progressive Era 
Unit 3 - The Nation at War 

PBL Activity - Wilson and the Paris Peace Conference: Constructing the 
Treaty of Versailles 

Unit 4 - Economic Expansion of the 1920s, The Depression, Franklin D. 
Roosevelt and the New Deal 

PBL Activity - Solving the Problems of the Depression: Constructing the 
New Deal 

Unit 5 - America and the World (1921-1945) 

PBL Activity - The Atomic Bomb: Truman’s Decision and Its Impact 

The Post War Era and Beyond -1945 to Present 

Unit 6 - The Cold War and Beyond 

Unit 7 - Civil Rights in the U.S.: Tracing Social, Economic, and Political 
Dynamics in the Last Half of the 20 th Century 
PBL Activity - The Issue of Affirmative Action: The Atlanta Case 

Unit 8 - Challenges of the New Century 

PBL Activity - Student Decisions on Current Issues: The Affordable 
Care Act; Immigration Reform; Debt, Spending, Taxes, and a Balanced 
Budget Amendment; Climate Change and Cap & Trade 
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Appendix B 

Metacognitive Reflection Questionnaire 

Please respond to each statement below by circling the number that best describes 

the thinking/reasoning you used during this activity. 

1 = Never (N) 2 = Rarely (R) 3 = Occasionally (S) 4 = Somewhat Often (SO) 

5 = Often (O) 6 = Very Often (VO) 

N R S SO O VO 

1. I used intuitive or emotional thinking (It felt right.) as I made a decision on this 
problem/issue. (Intuitive Thinking) 

1 2 3 4 5 6 

2.1 used logical-analytical thinking (application of logical operations) to develop 
a correct solution as I made a decision on this problem/issue. (Formal 
Thinking) 

1 2 3 4 5 6 

3. I considered context, multiple causes, multiple points of view, and conflicting 
ideas about what is true and relevant related to this problem/issue as I reached 
a conclusion or decision. (Postformal Thinking: Relativistic) 

1 2 3 4 5 6 

4.1 searched for and used inconsistencies and contradictions inherent in this 
problem/issue, sought to understand why those contradictory perspectives 
exist, and sought to resolve those contradictions as part of the problem-solving 
process. (Postformal Thinking: Dialectical) 

1 2 3 4 5 6 

5. I recognized that often there is no “correct” answer when dealing with complex 
problems/issues like this one. (Postformal Thinking: Dialectical) 

1 2 3 4 5 6 
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6.1 considered how change could affect this problem/issue and possibly my 
opinion/decision. (Postformal Thinking: Dialectical) 

1 2 3 4 5 6 

Please respond to the following question on the back of this questionnaire. 

7. Describe the various thinking systems you utilized during this problem-based 
activity, (from those listed above, and from more discipline specific processes 
like historical thinking, mathematical computation/estimation, etc.) How 
significant were these processes in helping you solve the problem or make a 
decision? 
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Appendix C 

Complex Postformal Thought (PTF) Questionnaire 

Please respond to each item below by circling the number that best describes you 

on the following scale: 1 = Not True (of self) and 7 = Very True (of self). 

1. I see the paradoxes in life (Paradoxes are inherent contradictions in reality.) 

1 2 3 4 5 6 7 

2.1 see more than one method that can be used to reach a solution or decision on a 
problem or issue. 

1 2 3 4 5 6 7 

3. I am aware that I can decide which reality or truth to experience at a particular 
time, but I know that reality and truth is really multi-level and more 
complicated. 

1 2 3 4 5 6 7 

4. There are many “right” ways to define any life experience; I must make a final 
decision on how I define the problems of life. 

1 2 3 4 5 6 7 

5.1 am aware that sometimes “succeeding” in the everyday world means finding 
a concrete answer to one of life’s problems, but sometimes it means finding a 
correct path that would carry me through any problems of this type. 

1 2 3 4 5 6 7 

6. Almost all problems can be solved by logic, but this may require different types 
of “logics.” 


1 2 3 4 5 6 7 

7.1 tend to see several causes connected with any event. 

1 2 3 4 5 6 7 
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8. I see that a given dilemma always has several good solutions. 

1 2 3 4 5 6 7 

9. I realize that I often have several goals in mind, or that life seems to have 
several goals in mind for me. So I go toward more than one in following my 
path in life. 

1 2 3 4 5 6 7 

10.1 can see the hidden logic in others’ solutions to the problem of life, even if I 
don’t agree with their solutions and follow my own path. 

1 2 3 4 5 6 7 
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Appendix D 

End of Study Questionnaire 

Thank you for participating in this study. Please answer the 5 questions below 

based on your experience this semester in History 2112. 

1. Rank your level of engagement (active participation) in your History 2112 
course. (1 = not engaged, 5 = fully engaged). 

1 2 3 4 5 

Explain your response. 

2. How does your ranking of engagement in your 2112 course compare to other 
history courses you’ve taken (in college or high school)? 

Explain your response. 

3. Rank the level of relevancy of the content of this course. How relevant were the 
topics (content areas)? (1 = irrelevant, 5 = very relevant). 

Explain your response. 

4. Do you believe you have expanded your ability to think critically as a result of 
History 2112? If so, can you explain the how your thinking has changed and/or 
evolved? 

5. To what extent do you believe you may utilize the thinking skills you may have 
gained in History 2112 last semester as you continue your education and life in 
general? 

Explain your response. 


29 



Learning Communities Research and Practice, Vol. 2 [2014], Iss. 2, Art. 4 


Appendix E 

Operational Definitions: Closed Systems and Postformal 

Operational 

Closed Systems can be identified by the following characteristics: 

• Focus on a limited number of aspects to the exclusion of other potentially 
useful aspects 

• Use of a familiar problem solving framework 

• Expectation of a single right answer applicable to all similar circumstance 

Analytical/Formal (Closed Systems) Problem-Solving - A closed systems 
problem-solver will generally apply a practiced systematic/formal problem¬ 
solving framework based on previous experience with similar problems to solve 
the problem at hand. This framework involves a problem-solving dynamic based 
on a limited number of variables, with other important aspects of the problem 
often judged as irrelevant to the solution. This causes formal thinkers to expect to 
produce a single right answer that will apply to all similar circumstances. 

People are using Postformal Problem-Solving when they recognize some or all of 
the following: 

• What is considered to be true can change when perspective and context 
change 

• Contradictions are often critical to understanding the complexities of a 
problem 

• The sides surrounding contradictions are interrelated and must be 
accommodated in the development of resolution alternatives 

• Some problems and issues don’t have simple and neat solutions 

• Even though a solution to a messy, real-life problem is identified, there 
will always be new challenges as the world and the people involved 
change. 

Relativistic thinkers recognize that, as a person’s perspective or context changes, 
so too does her/his perspective on what is true. Rather than seeing problems and 
issues through the lens of fixed truths and/or good vs. bad, relativistic thinkers 
recognize that context and contradictions are key to understanding the 
complexities of a problem, developing potential resolution alternatives, and 
recognizing the fact that no resolution may be possible for some problems and 
issues. Dialectical thinking involves the integration of relativistic thinking with 
the recognition that both sides of contradictions within a problem/issue are not 
only interrelated and connected but are also critical in the development of 
resolution alternatives. Inconsistencies and contradictions within problems and 
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issues become catalysts in the application of multiple cognitive systems as 
dialectical thinkers seek resolutions that lead to higher levels of understanding 
and cognition. Dialectical thinkers also recognize that any resolution or stability 
that may result from dialectical problem-solving operations will be perpetually 
challenged by new challenges, changes, and a potential tension to resolution to 
tension dynamic. 
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